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Abstract. Proving programs terminating is a fundamental computer 
science challenge. Recent research has produced powerful tools that can 
check a wide range of programs for termination. The analog for prob- 
abilistic programs, namely termination with probability one ("almost- 
sure termination"), is an equally important property for randomized 
algorithms and probabilistic protocols. We suggest a novel algorithm 
for proving almost-sure termination of probabilistic programs. Our algo- 
rithm exploits the power of state-of-the-art model checkers and termi- 
nation provers for nonprobabilistic programs: it calls such tools within 
a refinement loop and thereby iteratively constructs a "terminating pat- 
tern", which is a set of terminating runs with probability one. We report 
on various case studies illustrating the effectiveness of our algorithm. 
As a further application, our algorithm can improve lower bounds on 
reachability probabilities. 



1 Introduction 

Proving program termination is a fundamental challenge of computer science. 
Termination is expressible in temporal logic, and so checkable in principle by 
LTL or CTL model-checkers. However, recent research has shown that special 
purpose tools, like Terminator and ARMC |17I4) . and techniques like transition 
invariants, can be dramatically more efficient I16I19I18] . 

The analog of termination for probabilistic programs is termination with prob- 
ability one, or almost sure termination, abbreviated here to a. s. -termination. 
Since a. s. -termination is as important for randomized algorithms and probabilis- 
tic protocols as termination is for regular programs, the question arises whether 
the very strong advances in automatic termination proving termination can be 
exploited in the probabilistic case. However, it is not difficult to see that, with- 
out further restricting the question, the answer is negative. The reason is that 
termination is a purely topological property of the transition system associated 
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to the program, namely absence of cycles, but a. s. -termination is not. Consider 
for instance the program 

k = 1; while (0 < k) { if coin(p) k++ else k— } 

where coin(p) yields 1 with probability < p < 1, and with probability 
(1 — p). The program has the same executions for all values of p (only their 
probabilities change), but it only terminates a.s. for p < 1/2. This shows that 
proving a.s. -termination requires arithmetic reasoning not offered by termination 
provers. 

The situation changes if we restrict our attention to weakly finite probabilis- 
tic programs. Loosely speaking, a program is weakly finite if the set of states 
reachable from any initial state is finite. Notice that the state space may be 
infinite, because the set of initial states may be infinite. Weakly finite programs 
are a large class, which in particular contains parameterized probabilistic pro- 
grams, i.e., programs with parameters that can be initialized to arbitrary large 
values, but are finite-state for every valuation of the parameters. One can show 
that a.s. -termination is a topological property for weakly finite programs. If the 
program is deterministic, then it terminates a.s. iff for every reachable state s 
there is a path in the non-probabilistic program obtained by making all prob- 
abilistic choices nondeterministic leading from s to a terminating state, which 
corresponds to the CTL property AG EF end. 

(In the nondeterministic case there is also a corresponding topological property.) 
As in the nonprobabilistic case, generic infinite-state model checkers perform 
poorly for these properties because of the quantifier alternation AG EF. In par- 
ticular, CEGAR approaches usually fail, because, crudely speaking, they tend 
to unroll loops, which is essentially useless for proving termination. 
In [1], Arons, Pnueli and Zuck present a different and very elegant approach 
that reduces a.s. -termination of a probabilistic program to termination of a 
nondeterministic program obtained with the help of a Planner. A Planner oc- 
casionally and infinitely often determines the outcome of the next k random 
choices for some fixed k, while the other random choices are performed nonde- 
terministically. In this paper we revisit and generalize this approach, with the 
goal of profiting from recent advances on termination tools and techniques not 
available when [1] was published. While we also partially fix the outcome of 
random choices, we do so more fiexibly with the help of patterns. A first advan- 
tage of patterns is that we are able to obtain a completeness result for weakly 
finite programs, which is not the case for Planners. Further, in contrast to [1], 
we show how to automatically derive patterns for finite-state and weakly finite 
programs using an adapted version of the CEGAR approach. Finally, we apply 
our a.s. -termination technique to improve CEGAR-algorithms for quantitative 
probabilistic verification |6I7I9I5| . 

In the rest of this introduction we explain our approach by means of examples. 
First we discuss finite-state programs and then the weakly finite case. 

Finite-state programs. Consider the finite-state program FW shown on the left of 
Fig.[TJ It is an abstraction of part of the Fire Wire protocol Jlj. Loosely speaking. 
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k = 0; 



cl = ?; c2 = 2; 
k = 0; 

while (k < 100) { 



while (k < 100) { 
old_x = x; 
X = coin(p) ; 
if (x != old_x) k++ 



old_x = x; 

if (cl > 0) {x = nondetO; cl— } 

elseif (c2 = 2 ) { X = 0; c2— } 

elseif (c2 = 1 ) { X = 1; c2— } 

else /* cl = and c2 = */ {cl = ?; c2 = 2}> 



} 



if (x != old_x) k++ 

} 



Fig. 1. The programs FW and FW. 



FW terminates a.s. because if we keep tossing a coin then with probability 1 we 
observe 100 times two consecutive tosses with the opposite outcome (we even 
see 100 times the outcome 01). More formally, let C = {0, 1}, and let us identify 
a run of FW (i.e., a terminating or infinite execution) with the sequence of O's 
and I's corresponding to the results of the coin tosses carried out during it. 
For instance, (01)^^ and (001100)^° are terminating runs of FW, and 0" is a 
nonterminating run. FW terminates because the runs that are prefixes of (C*01)" 
have probability 1, and all of them terminate. But it is easy to see that these 
are also the runs of the nondeterministic program FW' on the right of Fig. [T] 
where c = ? nondeterministically sets c to an arbitrary nonnegative integer. 
Since termination of FW' can easily be proved with the help of ARMC, we have 
proved a.s. -termination of FW. 

Our reasoning is based on the following simple proof rule, with P a probabilistic 
program and R a set of runs of P: 



We present an automatic procedure leading from FW to FW ' based on the notion 
of patterns. A pattern is a subset of C" of the form C*wiC*W2C*W3 . . ., where 
■u;i , zi;2 , • . • G C* . We call a pattern simple if it is of the form (C*w)". A pattern 
is terminating (for a probabilistic program P) ii all runs of P that conform to 
i.e., that are prefixes of words of terminate. In the paper we prove the following 
theorems: 

(1) For every pattern (p and program P, the (^-conforming runs of P have prob- 
ability 1. 

(2) Every finite-state program has a simple terminating pattern. 

By these results, we can show that FW terminates a.s. by finding a simple termi- 
nating pattern (p, taking for P' a nondeterministic program whose runs are the 
^-conforming runs of P, and proving that P' terminates. In the paper we show 
how to automatically find (P with the help of a finite-state model-checker (in our 
experiments we use SPIN). We sketch the procedure using FW as example. First 
we check if some run of FW conforms to <?o = C'^ , i-e., if some run of FW is infi- 
nite, and get vi — 0" as answer. Using an algorithm provided in the paper, we 



Pr[R] = 1 



Every r S i? is terminating 



P terminates a.s. 
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compute a spoiler wi of ui : a finite word that is not an infix of vi . The algorithm 
yields wi = 1. We now check if some run of FW conforms to = {C*wi)'^ , and 
get V2 = 1'^ as counterexample, and construct a spoiler W2 of both vi and V2: 
a finite word that is an infix of neither v'^ nor v^. We get W2 = 01, and check 
if some run of FW conforms to ^2 = (C*u'2)"- The checker finds no counterex- 
amples, and so ^2 is terminating. In the paper we prove that the procedure is 
complete, i.e., produces a terminating pattern for any finite-state program that 
terminates a.s. 

Weakly finite programs. We now address the main goal of the paper: proving 
a.s. -termination for weakly finite programs. Unfortunately, Proposition (2) no 
longer holds. Consider the random- walk program RW on the left of Fig. [21 where 
N is an input variable. RW terminates a.s., but we can easily show (by setting N 

K = 2; cl = ?; c2 = K; 
k = 1 

while (0 < k < N) ■[ 
if (cl > 0) { 

if nondetO k++ else k — ; cl — 

}; 

elseif (c2 > 0) {k— ; c2— } 
else {K++; cl = ?; c2 = K} 

} 

Fig. 2. The programs RW and RW 

to a large enough value) that no simple pattern is terminating. However, there is 
a terminating pattern, namely <P = C*OOC*OOOC*0000 . . .: every (^-conforming 
run terminates, whatever value N is set to. Since, by result (1), the (^-conforming 
runs have probability 1 (intuitively, when tossing a coin we will eventually see 
longer and longer chains of O's), RW terminates a.s. In the paper we show that 
this is not a coincidence by proving the following completeness result: 

(3) Every weakly finite program has a (not necessarily simple) terminating pat- 
tern. 

In fact, we even prove the existence of a universal terminating pattern, i.e., a 
single pattern such that for all weakly finite, a.s. -terminating probabilistic 
programs all <?„-conforming runs terminate. This gives a universal reduction of 
a.s. -termination to termination, but one that is not very useful in practice. In 
particular, since the universal pattern is universal, it is not tailored towards 
making the proof of any particular program simple. For this reason we propose 
a technique that reuses the procedure for finite-state programs, and extends it 
with an extrapolation step in order to produce a candidate for a terminating 
pattern. We sketch the procedure using RW as example. Let RW^ be the program 
RW with N = i. Since every RW^ is finite-state, we can find terminating patterns 



k = 1; 

while (0 < k < N) { 

if coin(p) k++ else k — 

} 
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<Pi — {C*Ui)'^ for a finite set of values of i, say for i — 1,2,3,4,5. We obtain 
ui — U2 — e, U3 — 00, 7i4 — 000, ii5 — 000. We prove in tlie paper that <Pi is 
not only terminating for RW^, but also for every RWj with j < i. This suggests to 
extrapolate and take the pattern <P — C*00C*000C*0000 ... as a candidate for 
a terminating pattern for RW. We automatically construct the nondetcrministic 
program RW ' on the right of Fig. [2l Again, ARMC proves that RW ' terminates, 
and so that RW terminates a.s. 

Related work. A.s. -termination is highly desirable for protocols if termination 
within a fixed number of steps is not feasible. For instance, [3j considers the 
problem of reaching consensus within a set of interconnected processes, some of 
which may be faulty or even malicious. They succeed in designing a probabilistic 
protocol to reach consensus a.s., although it is known that no deterministic al- 
gorithm terminates within a bounded number of steps. A well-known approach 
for proving a.s. -termination are Pnueli et al.'s notions of extreme fairness and 
a-fairness [14ll5j . These proof methods, although complete for finite-state sys- 
tems, are hard to automatize and require a lot of knowledge about the con- 
sidered program. The same applies for the approach of Mclver et al. in [10] 
that offers proof rules for probabilistic loops in pGCL, an extension of Dijk- 
stra's guarded language. The paper [T^] discusses probabilistic termination in 
an abstraction-interpretation framework. It focuses on programs with a (single) 
loop and proposes a method of proving that the probability of taking the loop k 
times decreases exponentially with k. This implies a.s. -termination. In contrast 
to our work there is no tool support in [12 . 

Organization of the paper. Sections 2 contains preliminaries and the syntax and 
semantics of our model of probabilistic programs. Section 3 proves soundness 
and completeness results for termination of weakly finite programs. Section 4 de- 
scribes the iterative algorithm for generating patterns. Section 5 discusses case 
studies. Section 6 concludes. For space reasons, a full discussion of nondetcr- 
ministic programs and some missing proofs have been moved to an appendix. A 
shorter version of this paper will appear in the proceedings of the 24th Computer 
Aided Verification conference (CAV 2012). 

2 Preliminaries 

For a finite nonempty set E, we denote by E* and E'^ the sets of finite and 
infinite words over E, and set E°° = E* U E'^ . 

Markov Decision Processes and Markov chains. A Markov Decision Pro- 
cess (MDP) is a tuple M = (Qa, Qp, Init, — Lab^i, Labp), where Qa and Qp 
are countable or finite sets of action nodes and probabilistic nodes, Init C Qa^Qp 
is a set of initial nodes, and Lab^ and Labp are disjoint, finite sets of action 
labels and probabilistic labels. Finally, the relation — s- is equal to — 5-a U — 
where -^a ^ Qa x Lab^ x {Qa U Qp) is a set of action transitions, and 
— >p C Qp X (0, 1] X Labp x Q is a set of probabilistic transitions satisfying the 
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Fig. 3. Example MDP. 

following conditions: (a) if {q,p, I, q') and ((Z,p', q') are probabilistic transitions, 
then p = p'; (b) the probabilities of the outgoing transitions of a probabilistic 
node add up to 1. We also require that every node of Qa has at least one suc- 
cessor in -^A- If Qa = and Init = {qi} then we call A4 a Markov chain and 
write M = {Qp, qi, — >, Labp). 

We set Q = Qa^Qp and Lab — Lab^iULabp. We write q ^ q' for {q, I, q') e -^a, 
and q q' for {q,p, l,q') £ — >p (we skip p if it is irrelevant). For w — I1I2 ■ ■ ■ In & 
Lab*, we write q ^ q' ii there exists a path q — qo — ^ 91 — • • • --^ qn = q' ■ 

Example 1. Figure [3] shows an example of a Markov Decision Process A4 = 
({ga}, {91, 92, (Js}, Init, — ?>, Lab^, Labp), with action labels ao,ai, probabilistic 
labels T, co,ci, and a single initial node qa- 




Runs, paths, probability measures, traces. A run of an MDP M is an 

infinite word r — qoloqih ... £ (QLab)" such that for all i > either qi q^+i 

for some p G (0, 1] or qi qi+i- We call the run initial if qo G Init. We denote 
the set of runs starting at a node q by Runs''^ (q) , and the set of all runs starting 
at initial nodes by Runs(7W). 

A path is a proper prefix of a run. We denote by Paths''^ (g) the set of all paths 

starting at q. We often write r = go ^ 91 ^ 92 ^ • • • instead of r = qalaqi . . . 
for both runs and paths, and skip the superscripts of Runs(-) and Paths(-) if the 
context is clear. 

We take the usual, cylinder-based definition of a probability measure Pr^g on the 
set of runs of a Markov chain A4 starting at a state qo £ Init (see e.g. [2] or the 
appendix) for details). For general MDPs, we define a probability measure Pr^^ 
with respect to a strategy S. We may drop the subscript if the initial state is 
irrelevant or understood. 

The trace of a run r ~ qo qi . . . G Runs(A^), denoted by f, is the 
infinite sequence aoai . . . G Lab of labels. Given S C Lab, we define f\x; as the 
projection of r onto S. Observe that r\s can be finite. 

2.1 Probabilistic Programs 

We model probabilistic programs as flowgraphs whose transitions are labeled 
with commands. Since our model is standard and very similar to [9] , we give an 
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informal but hopefully precise enough definition. Let Var be a set of variable 
names over the integers (the variable domain could be easily extended), and let 
Val be the set of possible valuations of Var, also called configurations. The set 
of commands contains 

— conditional statements, i.e., boolean combinations of expressions e < e', 
where e, e' are arithmetic expressions (eg, x + y < 5 A y > 3); 

— deterministic assignments x := e and nondcterministic assignments x := 
nondet() that nondcterniinistically assign to x the value or 1; 

— probabilistic assignments x := coin(p) that assign to x the value or 1 with 
probability p oi {1 — p), respectively. 

A probabilistic program P is a tuple (£, /, label, ±, T), where £ is a finite set 
of control fiow locations, I C Val is a set of initial configurations, ^ C C x C is 
the flow relation (as usual we write I /' for (l, I') e and call the elements 
of ^ edges), label is a function that assigns a command to each edge, ± is the 
start location, and T is the end location. The following standard conditions must 
hold: (i) the only outgoing edge of T is T ^ T; (ii) either all or none of the 
outgoing edges of a location are labeled by conditional statements; if all, then 
every configuration satisfies the condition of exactly one outgoing edge; if none, 
then the location has exactly one outgoing edge; (Hi) if an outgoing edge of a 
location is labeled by an assignment, then it is the only outgoing edge of this 
location. 

A location is nondcterministic if it has an outgoing edge labeled by a nondc- 
terministic assignment, otherwise it is deterministic. Deterministic locations can 
be probabilistic or nonprobabilistic. A program is deterministic if all its locations 
are deterministic. 

Program Semantics. The semantics of a probabilistic program is an MDP. 

Let P be a probabilistic program (£,/, label, _L, T), and let jCd,£^a denote 
the sets of deterministic and nondcterministic locations of P. The semantics of 
P is the MDP Mp := (Qa, Qn, Init, Lab^, Labp), where Qa = Ca x Val is 
the set of nondeterministic nodes, Qd — {{C- \ Ca) x Val) U {T} is the set of 
deterministic nodes, Init — {_L} x / is the set of initial nodes, Lab^ = {ao,ai} 
is the set of action labels, Labp = {r, 0, 1} is the set of probabilistic labels, and 
the relation is defined as follows: For every node v = {l,a) oi Mp and every 
edge Z M- /' of P 

— if label(Z,/') = {x := coin(p)), then v — ^ {l',cr[x i->- 0]) and v — > 

{l',a[x^l]); 

— if label(Z,/') = {x := nondet()), then v {l',a[x h-> 0]) and v 
{l',a[x ^ 1]); 

T 1 

— if la,hel{l,l') = (x := e), then v — ^ {l',a[x — >■ e{a)]), where cr[x -> e{a)] 
denotes the configuration obtained from a by updating the value of x to the 
expression e evaluated under a; 

T 1 

— if label(i, I') = c for a conditional c satisfying a, then v — ^ {I', a). 
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For each node v = (T, cr), u — T and T — > T. □ 
A program P — (£, /, label, _L, T) is called 

— a. s. -terminating if Pr^[{r G Runs(A^p) | r reaches T}] — 1 for every strat- 
egy S and every initial state g of A^p; 

— finite if finitely many nodes are reachable from the initial nodes of A^p; 

— weakly finite if Pf, is finite for all h ^ I, where Ph is obtained from P by 
fixing h as the only initial node. 

Assumption. We assume in the following that programs to be analyzed are de- 
terministic. We consider nondeterministic programs only in Section l3.ll 

3 Patterns 

We introduce the notion of patterns for probabilistic programs. A pattern re- 
stricts a probabilistic program by imposing particular sequences of coin toss 
outcomes on the program runs. For the rest of the section we fix a prob- 
abilistic program P = (£, /, label, ±, T) and its associated MDP Mp = 
{Qa, Qp, Init, Labyi, Labp). 

We write C :— {0, 1} for the set of coin toss outcomes in the following. A pattern 
is a subset of of the form C*wiC*W2C*W3 . .., where wi,W2, ■ ■ ■ € S* . We say 
the sequence wi,'W2, ■ ■ ■ induces the pattern. Fixing an enumeration xi,X2, ■ ■ ■ 
of C*, we call the pattern induced by xi,X2t--- the universal pattern. For a 
pattern (p, a run r G Runs(A^p) is <!>- conforming if there is u G ^ such that f\c 
is a prefix of v. We call a pattern terminating (for P) ii all ^-conforming runs 
terminate, i.e., reach T. We show the following theorem: 

Theorem 2. 

(1) Let <P he a pattern. The set of <1>- conforming runs has probability 1. In par- 
ticular, if is terminating, then P is a. s. -terminating. 

(2) If P is a. s. -terminating and weakly finite, then the universal pattern is ter- 
minating for P. 

(3) If P is a. s. -terminating and finite with n < oo reachable nodes in A4p, then 
there exists a word w € C* with \w\ G 0{n^) such that C*wC" is terminating 
for P. 

Part (1) of Theorem [5] is the basis for the pattern approach. It allows to ignore 
runs that are not conforming, because they have probability 0. Part (2) states 
that the pattern approach is "complete" for a. s. -termination and weakly finite 
programs: For any a. s. -terminating and weakly finite program there is a termi- 
nating pattern; moreover the universal pattern suffices. Part (3) refines part (2) 
for finite programs: there is a short word such that C*wC^ is terminating. 

Proof (of Theorem\^. 

Part (1) (Sketch): We can show that the set of runs r that visit infinitely many 
probabilistic nodes and do not have the form C*wiC'^ is a null set. This result 
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Fig. 4. Nondeterministic a.s. -terminating program without terminating pattern. 

can then easily be generalized to C*wiC*W2 ■ ■ .C*w„C". All runs conforming 
<P can then be formed as a countable intersection of such run sets. 
Part (2): Let cri,CT2,... be a (countable or infinite) enumeration of the nodes 
in /. With Part (3) we obtain for each i > 1 a word Wi such that C*WiC'^ is 
a terminating pattern for P, if the only starting node considered is ct;. By its 
definition, the universal pattern is a subset of C*WiC^ for every i > 1, so it is 
also terminating. 

Part (3) (Sketch): Since P is a.s. -terminating, for every node q there exists a 
coin toss sequence Wq, \wq\ < n, with the following property: a run that passes 
through q and afterwards visits exactly the sequence Wq of coin toss outcomes is 
terminating. We build a sequence w such that for every state q every run that 
passes through q and then visits exactly the sequence w is terminating. We start 
with w = Wq for an arbitrary q T. Then we pick a. q' ^ T such that for q" ^ q, 
runs starting in q" and visiting exactly the probabilistic label sequence w lead 
to q'. We set w — WqWqi; after visiting w, all runs starting from q and q' end 
in T. We iterate this until no more q' can be found. We stop after at most n 
steps and obtain a sequence w of length < n^. □ 

3.1 Nondeterministic Programs 

For nondeterministic a.s. -terminating programs, there might not exist a termi- 
nating pattern, even if the program is finite. Figure |4] shows an example. Let <P 
be a pattern and C1C2C3 . . . G The run 

in A^P is (^-conforming but nonterminating. 

We show that the concept of patterns can be suitably generalized to nondeter- 
ministic programs, recovering a close analog of Theorem [21 Assume that the 
program is in a normal form where nondeterministic and probabilistic locations 
strictly alternate. This is easily achieved by adding dummy assignments. Writing 
A := {ao,ai}, every run r g Mp satisfies r\AuC & {AC)°° . 
A response of length n encodes a mapping A" — > C" in an "interleaved" fashion, 
e.g., {aol,aiO} is a response of length one, {ooOaol, aoOiili oiOaol, aiOail} is 
a response of length two. A response pattern is a subset of (AC)" of the form 
(AC)* Ri{AC)* R2{AC)* . . ., where i?2, • ■ • are responses. If we now define 
the notions of universal and terminating response patterns analogously to the 
deterministic case, a theorem very much like Theorem [2] can be shown. For 
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instance, let ^ = (AC)* {qqI, a iO}{AC)'^ . Then every conforming run of the 
program in Fig. |4]has the form 

{±, ao) ^ . . . ^ q q J— ^> g" — > T — > . . . for an i G {0, 1}. 

This implies that the program is a. s. -terminating (for all strategies). See Ap- 
pendix]^ for the details. 

4 Our Algorithm 

In this section we aim at a procedure that, given a weakly finite program P, 
proves that P is a. s. -terminating by computing a terminating pattern. This 
approach is justified by Theorem [5] (1). In fact, the proof of Theorem [5] (3) 
constructs, for any finite a. s. -terminating program, a terminating pattern. How- 
ever, the construction operates on the Markov chain A4p, which is expensive 
to compute. To avoid this, we would like to devise a procedure which operates 
on P, utilizing (nonprobabilistic) verification tools, such as model checkers and 
termination provers. 

Theorem [2] (2) guarantees that, for any weakly finite a. s. -terminating program, 
the universal pattern is terminating. This suggests the following method for 
proving a. s. -termination of P: (i) replace in P all probabilistic assignments by 
nondeterministic ones and instrument the program so that all its runs are con- 
forming to the universal pattern (this can be done as we describe in Section 14.11 
below); then (ii) check the resulting program for termination with a termina- 
tion checker such as ARMC [17] . Although this approach is sound and complete 
(modulo the strength of the termination checker), it turns out to be useless in 
practice. This is because the crucial loop invariants are extremely hard to catch 
for termination checkers. Already the instrumentation that produces the enu- 
meration of C* requires a nontrivial procedure (such as a binary counter) whose 
loops are difficult to analyze. 

Therefore we devise in the following another algorithm which tries to compute a 
terminating pattern C*wiC*W2 ... It operates on P and is "refinement" -based. 
Our algorithm uses a "pattern checker" subroutine which takes a sequence 
Wi,W2, ■ ■ and checks (or attempts to check) whether the induced pattern is 
terminating. If it is not, the pattern checker may return a lasso as counterexam- 
ple. Formally, a lasso is a sequence 

7717 ^rn ) ^ ... ^ {In, CTn) with Cr„) (/ 

and (/i, (Ti) e Init. We call the sequence {Im, fm) —>...—!> {ImOn) the lasso loop 
of the lasso. Note that a lasso naturally induces a run in Runs (Alp). If P is 
finite, pattern checkers can be made complete, i.e., they either prove the pattern 
terminating or return a lasso. 

We present our pattern-finding algorithms for finite-state and weakly finite pro- 
grams. In Section [4. II we describe how pattern-finding and pattern-checking can 
be implemented using existing verification tools. 
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Finite Programs. First we assume that the given program P is finite. The 
algorithm may take a base word sq G C* as input, which is set to sq ■— £ by 
default. Then it runs the pattern checker on C*sqC*sq ... If the pattern checker 
shows the pattern terminating, then, by Theorem [5] (1), P is a. s. -terminating. 
Otherwise the pattern checker provides a lasso (Zi,cri) — > . . . — s> {lm,o'rn) ^ 
{Im'^n)- Our algorithm extracts from the lasso loop a word ui € C*, 
which indicates a sequence of outcomes of the coin tosses in the lasso loop. If ui = 
e, then the pattern checker has found a nontcrminating run with only finitely 
many coin tosses, hence P is not a. s. -terminating. Otherwise (i.e., ui ^ e), let 
Si G C* be a shortest word such that sq is a prefix of si and si is not an infix 
of uf. Our algorithm runs the pattern checker on C*siC*si ... If the pattern 
checker shows the pattern terminating, then P is a. s. -terminating. Otherwise 
the pattern checker provides another lasso, from which our algorithm extracts 
a word U2 G C* similarly as before. If U2 = e, then P is not a. s. -terminating. 
Otherwise, let S2 G C* be a shortest word such that sq is a prefix of S2 and S2 
is neither an infix of nor an infix of U2 . Observe that the word si is an infix 
of U2 by construction, hence S2 7^ si- Our algorithm runs the pattern checker 
on C* S2C* S2 ■ ■ ■ and continues similarly, in each iteration eliminating all lassos 
so far discovered. 

The algorithm is complete for finite and a. s. -terminating programs: 

Proposition 3. Let P be finite and a. s. -terminating. Then the algorithm finds 
a shortest word w such that the pattern C*wC*w . . . is terminating, thus proving 
termination of P. 

In each iteration the algorithm picks a word Sj that destroys all previously 
discovered lasso loops. If the loops are small, then the word is short: 

Proposition 4. We have \sj\ < |so| + 1 + log2 + • ■ • + \uj\). 

The proofs for both propositions can be found inAppendix IB. 21 

Weakly Finite Programs. Let us now assume that P is a. s. -terminating and 
weakly finite. We modify our algorithm. Let hi,h2, ■ ■ ■ be an enumeration of 
the set / of initial nodes. Our algorithm first fixes hi as the only initial node. 
This leads to a finite program, so we can run the previously described algo- 
rithm, yielding a word wi such that C*wiC*wi ... is terminating for the initial 
node hi. Next our algorithm fixes 62 as the only initial node, and runs the pre- 
viously described algorithm taking wi as base word. As before, this establishes 
a terminating pattern C*W2C*W2 . . ■ By construction of W2, the word wi is a 
prefix of W2, so the pattern C*wiC*W2C*W2 ■ . ■ is terminating for the initial 
nodes {&i,&2}- Continuing in this way we obtain a sequence wi,u;2,... such 
that C*wiC*W2 ... is terminating. Our algorithm may not terminate, because 
it may keep computing wi,W2, . ■ .■ However, we will illustrate that it is promis- 
ing to compute the first few Wi and then guess an expression for general Wi. 
For instance if wi = and W2 = 00, then one may guess Wi = 0\ We encode 
the guessed sequence wi,W2,... in a finite way and pass the obtained pattern 
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true 




c = C3 



c = 



Cn-l C = C„ 

c = 2 



true c = 2 c = 2 



Fig. 5. Biichi automaton ^(lu), for w = ciC2...c„ € C*. Note that the number of 
states in Alw) grows hnearly in \'w\. 

C*wiC*W2 ... to a pattern checker, which may show the pattern terminating, 
estabhshing a. s. -termination of the weakly finite program P. 

4.1 Implementing Pattern Checkers 

Finite Programs. We describe how to build a pattern checker for finite pro- 
grams P and patterns of the form C*'wC*w . . . We employ a model checker 
for finite-state nonprobabilistic programs that can verify temporal properties: 
Given as input a finite program and a Biichi automaton A, the model checker 
returns a lasso if there is a program run accepted by A (such runs are called 
"counterexamples" in classical terminology) . Otherwise it states that there is no 
counterexample. For our case studies, we use the SPIN tool [8]. 
Given a finite probabilistic program P and a pattern = C*wC*w . . ., we first 
transform P into a nonprobabilistic program P' as follows. We introduce two 
fresh variables c and term, with ranges {0, 1, 2} and {0, 1}, respectively, and add 
assignments term := and term := 1 at the beginning and end of the pro- 
gram, respectively. Then every location I of P with label{l^ I') = x := coin(p) for 

a label /' is replaced by a nondeterministic choice and an if-statement as follows: 
X : = nondet ; 

if (x = 0) c := 0; c := 2; else c := 1; c := 2; end if; 

In this way we can distinguish coin toss outcomes in a program trace by inspect- 
ing the assignments to c. Now we perform two checks on the nonprobabilistic 
program P': 

First, we use SPIN to translate the LTL formula G ^term A FG{c ^ {0, 1}) into 
a Biichi automaton and check whether P' has a run that satisfies this formula. 
If there is indeed a lasso, our pattern checker reports it. Observe that by the 
construction of the LTL formula the lasso encodes a nonterminating run in P 
that eventually stops visiting probabilistic locations. So the lasso loop does not 
contain any coin tosses (and our algorithm will later correctly report that P is 
not a. s. -terminating). Otherwise, i.e., if no run satisfies the formula, we know 
that all nonterminating runs involve infinitely many coin tosses. Then we perform 
a second query: We construct a Biichi automaton Aiw) that represents the set 
of infinite (^-conforming runs, see Fig. [5] We use SPIN to check whether P' 
has run that is accepted by A{w). If yes, then there is an infinite (^-conforming 
run, and our pattern checker reports the lasso. Otherwise, it reports that is a 
terminating pattern. 



13 



X := nondetO ; 
if (ctr <= 0) 

if (pos > length (w [next] ) ) ctr := ?; pos := 1; next := next+1; 
else x := w [next] [pos] ; pos := pos+1; 
else ctr := ctr-1; 

Fig. 6. Code transformation for coin tosses in weakly finite programs. 



Weakly Finite Programs. Recall that for weakly finite programs, the pat- 
tern checker needs to handle patterns of a more general form, namely ^ = 
C*wiC*W2 . ■ ■ Even simple patterns like C*OC*OOC*000 . . . cannot be repre- 
sented by a finite Biichi automaton. Therefore we need a more involved instru- 
mentation of the program to restrict its runs to (^-conforming ones. Now our 
pattern checker employs a termination checker for infinite-state programs. For 
our experiments we use ARMC. 

Given a weakly finite program P and a pattern = C*wiC*W2 ■ ■ ■, we trans- 
form P into a nonprobabilistic program as follows. We will use a command 
X := ?, which nondeterministically assigns a nonnegative integer to x. Further 
we assume that we can access the fc-th letter of the i-th element of (wi)ieN 
by w[i][k], and \wi\ by length(w[i]). We add fresh variables ctr, next and pos, 
where ctr is initialized nondeterministically with any nonnegative integer and 
next and pos are both initialized with 1. If a run r is ^-conforming, f\c is a 
prefix of viwiV2'W2V3W3 . . ., with Vi € C*. The variable ctr is used to "guess" 
the length of the words Vi; the individual letters in Vi are irrelevant. We replace 
every command c := coin(p) by the code sequence given in Fig. [6l 
The runs in the resulting program correspond exactly to the (^-conforming 
runs in P. Then is given to the termination checker. If it proves termination, 
we report is a terminating pattern for P" . Otherwise, the tool might either 
return a lasso, which our pattern checker reports, or give up on P^, in which 
case our pattern checker also has to give up. 

In our experiments, a weakly finite program typically has an uninitialized inte- 
ger variable N whose value is nondeterministically fixed in the beginning. The 
pattern C*wiC* . . . C*wnC'^ is then often terminating, which makes next < N 
an invariant in P*. The termination checker ARMC may benefit from this in- 
variant, but may not be able to find it automatically (for reasons unknown to the 
authors) . We therefore enhanced ARMC to "help itself" by adding the invariant 
next < to the program if ARMC's reachability mode can verify the invariant. 



5 Experimental evaluation 

We apply our methods to several parameterized programs taken from the liter- 
aturell 

^ The sources can be found at |http: //ww. model . in. tuiii.de/~gaiser/cav2012 .html! 
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Name 


#loc 


Pattern words for 


Time 


i-th word of 


Time 






iV = 1,2,3,4 


(SPIN) 


guessed pattern 


(ARMC) 


firewire 


19 


010 010 010 010 


17 sec 


010 


1 min 36 sec 


randomwalk 


16 


e 0^ 0^ 0" 


23 sec 


0" 


1 min 22 sec 


herman 


36 


010 0(10)2 0(10)^ 0(10)* 


47 sec 


0(10)' 


7 min 43 sec 


zeroconf 


39 


q3 q4 q5 q6 


20 sec 


0^+2 


26 min 16 sec 


brp 


57 


00 00 00 00 


19 sec 


00 


45 min 14 sec 



Fig. 7. Constructed patterns of the case studies and runtimes. 

— firewire: Fragment of FireWire's symmetry-breaking protocol, adapted 
from [11) (a simpler version was used in the introduction). Roughly speaking, 
the number 100 of Fig. [T]is replaced by a parameter N. 

— randomwalk: A slightly different version of the finite-range, one-dimensional 
random walk used as second example in the introduction. 

— herman: An abstraction of Herman's randomized algorithm for leader elec- 
tion used in |13) . It can be seen as a more complicated finite random walk, 
with N as the walk's length. 

— zeroconf: A model of the Zeroconf protocol taken from |9j. The protocol 
assigns IP addresses in a network. The parameter TV is the number of probes 
sent after choosing an IP address to check whether it is already in use. 

— brp: A model adapted from [9j that models the well-known bounded retrans- 
mission protocol. The original version can be proven a. s. -terminating with 
the trivial pattern C"; hence we study an "unbounded" version, where ar- 
bitrarily many retransmissions are allowed. The parameter TV is the length 
of the message that the sender must transmit to the receiver. 

Proving a.s. -termination. We prove a. s. -termination of the examples using 
SPIN [5 to find patterns of finite-state instances, and ARMC [T7] to prove 
termination of the nondeterministic programs derived from the guessed pattern. 
All experiments were performed on an Intel© 17 machine with 8GB RAM. The 
results are shown in Fig. [71 The first two columns give the name of the example 
and its size. The next two columns show the words wi, . . . ,Wi oi the terminating 
patterns C*wiC'^ , . . . ,C*WiC'^ computed for = 1,2,3,4 (see Theorem [2i;3) 
and Section [O]) . and SPIN's runtime. The last two columns give word Wi in the 
guessed pattern C*wiC*W2C*W3 . . . (see Section [4Tt . and ARMC's runtime. 
For instance, the entry 0(10)* for herman indicates that the guessed pattern is 
C*010C*01010C*0101010 . . .. 

We derive two conclusions. First, a.s. -termination is proved by very simple pat- 
terns: the general shape is easily guessed from patterns for TV = 1,2,3,4, and 
the need for human ingenuity is virtually reduced to zero. This speaks in fa- 
vor of the Planner technique of [1] and our extension to patterns, compared to 
other approaches using fairness and Hoare calculus [15|10) . Second, the runtime 
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is dominated by the termination tool, not by the finite-state checker. So the 
most direct way to improve the efficiency of our technique is to produce faster 
termination checkers. 

In the introduction we claimed that general purpose probabilistic model-checkers 
perform poorly for a. s. -termination, since they are not geared towards this prob- 
lem. To supply some evidence for this, we tried to prove a. s. -termination of the 
first four examples using the CEGAR-based PASS model checker |6|7j . In all 
four cases the refinement loop did not terminate^ 

Improving lower bounds for reachability. Consider a program of the 
form if coin(0.8) {PlO; else P2()}; ERROR . Probabilistic model-checkers 
compute lower and upper bounds for the probability of ERROR. Loosely speak- 
ing, lower bounds are computed by adding the probabilities of terminating runs 
of PI and P2. However, since CEGAR-based checkers |6|7|9|5j work with ab- 
stractions of PI and P2, they may not be able to ascertain that paths of the 
abstraction are concrete paths of the program, leading to poor lower bounds. 
Information on a. s. -termination helps: if e.g. PI terminates a.s., then we already 
have a lower bound of 0.8. We demonstrate this technique on two examples. The 
first one is the following modification of firewire: 

N = 1000; k = 0; miss = 0; 
while (k < N) { 

old_x = x; X = coin(0.5); 

if (x = old_x) k++ else if (k < 5) miss = 1 

} 

For i G {0, 1}, let pi be the probability that the program terminates with 
miss — i. After 20 refinement steps PASS returns upper bounds of 0.032 for po 
and 0.969 for pi, but a lower bound of for pi, which stays after 300 iter- 
ations. Our algorithm establishes that the loop a.s. -terminates, which implies 
Pa + Pi = 1, and so after 20 iterations we already get 0.968 < pi < 0.969. 
We apply the same technique to estimate the probabilities pi , pq that zeroconf 
detects/does-not-detect an unused IP address. For N — 100, after 20 refinement 
steps PASS reports an upper bound of 0.999 for po, but a lower bound of for pi, 
which stays for 80 more iterations. With our technique after 20 iterations we 
get 0.958 < pi < 0.999. 

6 Conclusions 

We have presented an approach for automatically proving a.s. -termination of 
probabilistic programs. Inspired by the Planner approach of [I], we instrument a 
probabilistic program P into a nondeterministic program P' such that the runs 
of P' correspond to a set of runs of P with probability 1. The instrumentation 

* Other checkers, like PRISM, cannot be applied because they only work for finite- 
state systems. 
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is fully automatic for finite-state programs, and requires an extrapolation step 
for weakly finite programs. We automatically check termination of P' profiting 
from new tools that were not available to [T . While our approach maintains the 
intuitive appeal of the Planner approach, it allows to prove completeness results. 
Furthermore, while in [I] the design of the Planner was left to the verifier, we 
have provided in our paper a CEGAR-like approach. In the case of parameterized 
programs, the approach requires an extrapolation step, which however in our 
case studies proved to be straightforward. Finally, we have also shown that our 
approach to improve the game-based CEGAR technique of [61719) for computing 
upper and lower bounds for the probability of reaching a program location. While 
this technique often provides very good upper bounds, the lower bounds are not 
so satisfactory (often 0), due to spurious nonterminating runs introduced by the 
abstraction. Our approach allows to remove the effect of these runs. 
In future work we plan to apply learning techniques to pattern generation, 
thereby inferring probabilistic termination arguments for large program in- 
stances from small instances. 
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balchenko for many discussions and their help with ARMC, and Bjorn Wachter 
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Appendix 

In Appendix |A] wc give details on patterns for nondeterministic programs. Ap- 
pendix [B] contains additional preliminaries that are needed for the following ap- 
pendices: In IB. II and IB. 21 proofs for Sections [3] respectively Section 2] are given. 
Appendix IB. 31 contains a proof for the theorem in Appendix |^ 

A Patterns for Nondeterministic Programs 

For general a. s. -terminating probabilistic programs, there might not exist a ter- 
minating pattern, even if the program is finite, recall Figured 
We therefore propose another pattern class that also takes nondeter- 
ministic decisions into account. We fix an arbitrary probabilistic pro- 
gram P = (£,/, label, _L, T), and its associated MDP A^p = 
(Qa, Qp, Init, — >, LabA, Labf). We assume that P is in a special normal form: 
Every nondeterministic location has a probabilistic location as its successor, ev- 
ery probabilistic location has a nondeterministic location as its successor. It is 
easy to transform a program in normal form by adding redundant probabilistic 
and nondeterministic locations such that the transformed program terminates 
iff the original one does. For example, the program in Fig. |4]is in normal form. 
If P is in normal form, then every run ?' G A^p is a prefix of a word in {AC)°°. 
We write A := {ao, ai} and G := {ao, ai} U C. A set C (AC)* is called a re- 
sponse of length n > if (i) every w G W has length 2n, (ii) for 'Wi,W2 & W with 
wi 7^ W2, wi\a ^ W2\a holds, and (iii) W contains exactly 2" elements. We de- 
note by Resp{n) the set of responses of length n, and set Resp := [J^^-^ Resp{n). 
Intuitively, a response R of length n contains for every sequence of nondeter- 
ministic actions of length n a sequence of coin toss outcomes of length n (inter- 
leaved in one word of R). For example, {aol,aiO} is a response of length one, 
{aoOaol, agOail, aiOaol, aiOail} is a response of length two. 
A response pattern is a subset of (ACY of the form {AC)* Ri{AC)* R2{AC)* . . ., 
where i?2, • • • are responses. We say i?2, . . . induces the response pattern. 
As in the deterministic case, fixing an enumeration i?i,i?2, • • ■ of all responses, 
we call the pattern [AC)* Ri{AC)* R2{AC)* a universal response pattern. For 
a response pattern a run r £ Runs(7Wp) is (^-conforming if there is w S 
such that fjc is a prefix of v. We call a response pattern ^ terminating if all 
^-conforming runs terminate. 

Analogously to the deterministic case, we show the following theorem in Ap- 
pendix IB. 31 

Theorem 5. Let P he a probabilistic program in normal form. 

(1) Let ^ be a response pattern. The set of (l>- conforming runs lias probability 1 
for every strategy S for A4p. Ln particular, if P has a terminating pattern, 
then P is a. s. -terminating. 

(2) If P is a. s. -terminating and weakly finite, then the universal pattern is ter- 
minating for P. 
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(3) If P is a. s. -terminating and finite with n < oo reachable nodes in Aip, then 
there exists a response R of length in 0{n^) such that (AC)* R{AC)'^ is 
terminating for P. 

B Proofs 
Preliminaries 

Let P = (£,/, label, _L, T), and let Mp he its corresponding MDP with 
Mp = {Qai Qd, Init, Lab^, Labp). 

The probability Pr[qi — 92] of a transition qi q2 is equal to p. We define a 
probability measure on the set of runs of a Markov chain A4 in the usual way (see 
e.g. [2]). The cylinder set Cyl(7r) of a path tt is the set of runs having tt as prefix. 
The probability of Cyl(7r) for a path tt starting at q, denoted by Prq[Cyl(7r)], is 
1 ii TT — q, and otherwise the product of the probabilities of the transitions of 
TT. There is a unique extension of Pr^ to a probability measure over the smallest 
(T-algebra &^ containing all cylinder sets starting at q. We denote it by Pr^. 
We say a set of runs is measurable if it is contained in &-^. 
For general MDPs, we can only define a probability measure after resolving 
the nondeterminism. A strategy for an MDP is a function S that maps the 

empty path e to an initial node qo, and every path go ^ <li-'-<ln-i -— > 
qn & Paths(7\/() ending at an action node to a probability distribution over the 

outgoing labels of g, i.e., over the labels I such that <?„ \ q for some node q. 
Given a strategy 5, we define the Markov chain A4[S'] as usual (see [2] for a 
formal definition): the nodes of A^[5'] are the paths of M. whose cylinders have 
nonzero probability, the transitions are defined to match the definition of the 
nodes, and the transition probabilities are assigned according to S. For every 
node (7 of we define a probability measure Pr^ over ©■'^, that assigns to a 
cylinder Cyl(7r) the probability of Cyl(7r') in the Markov chain A1[5'], where tt' 
is the unique path of M [S] starting at the node q (notice that q is also a path, 
and so a node of and ending at the node tt. We write Pr'^[-] for Prqj,[-]. 

B.l Proofs of Section [3] 

In this section we complete the proof of 

Theorem [2j Let P he a probabilistic program that is deterministic. 

(1) Let <P he a pattern. The set of (P- conforming runs has probability 1. In par- 
ticular, if <I? is terminating, then P is a. s. -terminating. 

(2) If P is a. s. -terminating and weakly finite, then the universal pattern is ter- 
minating for P. 

(3) If P is a. s. -terminating and finite with n < 00 reachable nodes in M.p, then 
there exists a word w & C* with |w| G 0{n'^) such that C*wC'^ is terminating 
for P. 
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Proof. 

It remains to prove parts (1) and (3). Let P = (£, 7, label, _L, T). Its corre- 
sponding MDP Alp is a Markov chain Mp = {Q,qi,^,Lahp). We write Pr[-\ 
instead of Prq^ [■]. Let # = C*iuiC*W2C* . . .; the set of runs conforming to ^ are 
denoted by Runs(<?). 

Proof of Part (1): 

We first prove that Runs(<?) is measurable. Let Ic G ©•'^^ be the set of runs r 
with f\c G C". For wi,W2, ■ ■ ■ Wi, i>l, we define the set S{wi,W2, • • • , Wi) by 

S{wi,W2, ...,Wi) := {r & Runs(Xp) | f|c G C*wiC*W2C* . . . C*w,C'^}. 

S{'w-i,'W2, ■ ■ ■ , Wi) is measurable: Let NC(i) G Q-^'' be the set of all runs r with 
the i-th label of r not in C, and F{i, c) G S-^J' the set of runs that have c as 
i-th label. Set 

G{b-,b+,ci...Ck)= U ( fl NC{l)n f] F{aj,cj)) 

6-<ai<...<afe<6+ l>ai ^<j<k 

Al^{ai,...,ak} 

for {b~,b^,k} C N and ci . . . c/c G C*^. S{wi,W2, ■ ■ ■ ,Wn) can be written as 

icn y G{b^,bt,wi)n...nG{b-,b+,Wn)e6^''. 

0<b- <b+ <b- <...<bn <bt 

Since 

Runs(^) = (Runs(Mp) \ /c) U p| S{wuW2, G 6^^ 

i>0 

we conclude that Runs(fp) is also measurable. 

Next we show that Pr[Runs(^)] = 1. 

For every prefix wi,W2, W3,. .. ,Wi of (u'i)jgN, Pr[S'('u;i, . . . , Wi)] = Pr[/c] holds, 
i.e., the set of runs that visit probabilistic nodes infinitely often, but are not 
C*wiC*W2C* . . . C*WiC"^-conforming, have probability zero. 
For proving this we write w = W1W2 ■ ■ -Wi. Let n = \w\. S{w) C 
S{wi,W2, ■ ■ ■ ,Wi) holds for all i. It suffices to show that Pr[6'(w)] = Pr[/c], 
since this implies with Ic 2 S{wi, . . . , Wi) that Pr[/(7] = Pr[S'(ii;i, u>2, . . . , Wi)]. 
Let V{j) be the set of runs that visit a probabilistic node at least j times, and 
let 

B{j) = V{j ■ n) n (Runs(Mp) \ S{w)) 

be the set of runs r that visit a probabilistic node at least j ■ n times, and w is 
no substring of f|c. 

Since there are only finitely many probabilistic locations in P, there exists a 
minimal probability Pmin > such that for every transition q q', c G {0, 1}, 
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v' ^ Pmin holds. We write NV(iy) ("not visited") for the set of runs r such that 
f\c does not start with w. Now 

Pr\B{\)\ 

< Pr[NV(w) I V{n)] ■ Pr[V{n)] 

<(1-Pmin), 

i.e., after visiting probabihstic nodes at least n times, the probability p of not 
seeing the sequence w is at most (1 — p'^^^) < 1- With a simple inductive argu- 
ment we obtain Pr[B{j)] < {1 - p]!^^^ . It holds that B{j) D B{j + 1) for all j. 
Then 

Prif] B{j)] = lim Pr[B{j)] < lim (1 - pJJ^)^ = 0. (1) 

We can write S{w) = Ic \ r\j>o ^ij)- Hence 

Pr[SiwiW2 . . . w,)] = Pr[Ic \ f] B{j)] (Def. of B{-)) 

^Pr[Ic]-Pr[f]B{j)nIc] 

^Pr[Ic] (Eq.HI). 
Now Pr[/c \ Siwi, . . .,Wi)] = Pr[Ic] ~ Fr[5(wi, . . . ,Wi)\ = 0. We can write 
-?^c \ Pi 5(^1, . . . , Wi) = -^c n y /c \ 5'(u;i, . . .,Wi). 

For every i > 1, /c \ S{wi, . . . ^Wi) is a null set, thus the countable union 
Ui>i -^c \ S{wi, . . . , Wi) is also a null set (*). 
We conclude: 

Pr[Runs(^)] 

= Pr[Runs(Xp) \ Ic] + Pr[f] S{wi, w,)] 

i>a 

= Pr[Rnns{M p) \ Ic] + Pr[Ic] (*) 
= 1. 

Proof of Part (3): 

We say that q E Q ends up in q' ^ Q following w — C1C2 ■ ■ ■ Cm 6 C* if 

t'^ Cit"^ C2t'^ ...T* CmT"' I 

q > q , 

and q' is probabilistic or T. Note that q' is unique if it exists, since P is deter- 
ministic. 
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For every reachable node q and every sequence w G C* holds that either: (i) q 
ends up in a node following w, or (ii) q ends up in T following a proper prefix of 
w. Otherwise there exists a reachable node q' ^ T from which no probabilistic 
location or T is reachable any more, which contradicts that P is a. s. -terminating. 
For every node q E Q, there exists a Wq C* such that q ends up in T following 
Wq, again due to the a. s. -termination property of P. We can choose Wq such that 
\wq\ < nhy removing cycles. 

We construct a sequence u;'^^^ u;'^^^ . . . , w'^™^ using the following algorithm. Set 
■u;(°) :— € and i :— 0. 

1. Pick a q,' e Q that does end up in a node qi ^ T following w^'^~^\ If no such 
qi exists set w := w^*"-'^' and terminate. 

2. Set := w^''~'^^Wq^. Set i := i + 1 and go to 1). 

The node sets Q^*' consist of T and all nodes a state q d Q might end up after 
following w^*). Q'*^ contains at most n — i nodes for every i >0. This is certainly 
true for i = 0. In the i-th iteration the chosen q'^ ends up in qi G Q*-*"^-* after 
following w^*"^^. After following Wq., qt ends up in T. Thus qi ends up in T after 
w^'^\ This implies that IQ^*-*] < IQ'-*^"'^-' |, since every node can end up in at most 
one node after following a nonempty coin sequence, and note that every Q^*^ 
contains T. 

After at most n — 1 iterations, < 1, and the algorithm terminates. Hence 

\w\ < (n — 1) • maxggQ \wq\ < {n — 1)^. Every node of Q ends up in T after 
following a prefix of w^*^. If it ended up in another node q, the algorithm would 
have performed another iteration, making w longer, if there were no node q' such 
that q ends up in g', a prefix of w must have led it to T before. 
We can conclude that every run r for which fjc is a prefix of a word C*wC'^ is 
terminating, and thus C*wC'^ is a terminating pattern. □ 

B.2 Proofs of Section lEl 

Proposition [3j Let P be finite and a. s. -terminating. Then the algorithm finds 
a shortest word w such that the pattern C*wC*w . . . is terminating, thus proving 
termination of P. 

Proof. Recall from the proof of Theorem [5] (3) that there is a fixed word z G C* 
which leads from an arbitrary node in Runs(A^p) to termination. In particular, 
z is never an infix of for any i. It follows that sqz is never an infix of for 
any i. Assume for a contradiction that our algorithm does not succeed in proving 
termination. Since the Si are all pairwise different, our algorithm eventually 
chooses Sj := sqz for some j G N. By the definition of z the pattern C*zC*z . . . 
is terminating, hence so is C*SjC*Sj ... It follows that the pattern checker shows 
Pj+i terminating, which is a contradiction. □ 

Proposition [4l We have \sj\ < |sol -I- 1 -I- log2 + • ■ • + \uj\). 
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Proof. If a word w is not an infix of any of tlie words uf , . . . , , then neither is 
sqw. Hence it suffices to construct such a word w with < 1 + log2 K, where 
K := |mi I + ■ • • + |mj I . Let pi , . . . , pK be an enumeration of all suffixes of the words 
u'^, . . . , u'j. For any word w, denote by S{'w) C {pi, . . . ,pk} the set of words 
p G {pi, . ■ . ,pk} such that w is a prefix of p. It suffices to construct w such that 
|w| < 1 + log2 K and S{w) — 0. We construct w iteratively. Let wq := e. In each 
iteration i, choose Wi+i := WiC with c £ {0,1} so that \S{'Wic)\ is minimized. 
Observe that 15(10^+1)1 < \S{wi)\/2, as all words in S{wi) start with either WiO 
or Wil. It follows that S (wi+i^iog, k\) — ^- ^ 

B.3 Proofs of Appendix lAl 

In this section we prove 

Theorem [5l Let P be a probabilistic program in normal form. 

(1) Let $ be a response pattern. The set of <!>- conforming runs has probability 1 
for every strategy S for A4p. In particular, if P has a terminating pattern, 
then P is a. s. -terminating. 

(2) If P is a. s. -terminating and weakly finite, then the universal pattern is ter- 
minating for P. 

(3) If P is a. s. -terminating and finite with n < oo reachable nodes in A4p, then 
there exists a response R of length in 0{n^) such that (AC)* R{AC)'^ is 
terminating for P. 

Proof. 

Let P = (£,/, label, _L, T). The MDP corresponding to P is denoted by 
Mp ^ (QA,(3D,Init,^,LabA,Labp). Let <P = (AC)* Ri{AC)* R2 . . with R, 
a response for alH > 1. We call the set of (^-corresponding runs Runs(<?). For 
responses Ri,R2 of length ui and 7^2, respectively, and a word w G (AC)+, we 
set woRi {wr \ r € Ri} and R10R2 ■= {rR2 \ r G Ri}. R10R2 is a response 
of length ni + 712. We set G := A U C . Recall that, since P is in normal form, 
for every run r in A^p, fjc is a prefix of a word in (AC)". 

Proof of Part (1): We first prove that Runs(<25) is measurable. Let Iq G Q^'' 
be the set of runs r with f\c^ G'^ . For Ri, R2, . . . Ri, i > 1, we define the set 
S{Rl,R2,...,R^) by 

S{Ri,R2,...,Rr) := {r G Runs(7Wp) | fjc G G* RiG* R2G* . . . G* R.,G'^}. 

S{Ri, R2, . . . , Ri) is measurable: The set of runs r such that f\M G 
G*wiG*W2G* . . .G*WiG'^ for {wi, ... ,Wi) e Ri x ... x Ri is measurable, which 
can be proved by an easy variation of the first part of the proof of Theorem [21 
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S{Ri,R2, . . . , Ri) is the finite union of aii tliese run sets and ttius is measurable. 
Again, since 

Runs(<?) = (Runs(Xp) \/g) U f] S{Ri, R2, . . . , R^) e ©"^^ 

i>0 

we conclude that Runs(<?) is measurable. 

Let 5 be a strategy for A4p. We show that Pr'^[Runs(<^)] = 1, again reusing 
ideas from the proof of Theoren [51 

For every prefix i?2, -R3, . . . , i?i of {Ri)ieN, we show that Pr[5(i?i, . . . , Ri)] = 
Pr[/G] holds, i.e., the set of runs that visit probabilistic nodes infinitely often, 
but are not conforming to (AC)* Ri{AC)* . . . (AC)* R.^iAC)'^ have probability 
zero. 

For proving this we write R ^ Rio R20 . . . R^. Let n be the length of R. S{R) C 
S{Ri,R2, .■.,Ri) holds for aU i. Again it suffices to show that Pr[S'(i?)] = Pr[/G], 
since this implies with Iq 3 S{Ri, . . . , Ri) that Pr[/G] ~ Pr[5(i?i, i?2, . • ■ , Ri)]- 
We reuse the definition of the sets of runs V{j) that visit a probabilistic node 
at least j times, and set 

B{j) = V{j ■ n) n (Runs(A^p) \ S{R)) 

the set of runs r that visit a probabilistic node at least j ■ n times, and no w ^ R 
is a substring of rjc. There exists a minimal probability Pmin > such that for 

every transition q — ^ q' in A^p[5], c e {0,1}, p' > Pmin holds. Note that this 
in general only holds for probabilistic transitions labeled by {0,1} in Mp[S]. 
For X ^ A* we write SC{x) ("strategy choice") for the set of runs r such that 
f\A starts with x. For x ^ x' with x, x' 6 A* having the same length, 

SC{x) n SC{x') = 0. (2) 

Let NV{w) ("not visited") be again the set of runs r such that f\c does not start 
with w £ C*. With this we get 

Pr^[B(l)] 

< Pr'^[^y{w\c) I SC{w\a) n V{n)] ■ Pr^[SC(wU) n V{n)] 
weR 

< J2{i-p':,,J-Pr'[sc{w\A)nvij)] (Eq.ED 

weR 

<(1-Pmi„)- 

Again we can see that after visiting probabilistic nodes at least n times, the 
probability of not seeing at least one of the w G i? is at most (1 — p'^i^) < 
1. In B{j), we repeat this experiment at least j times; by a simple inductive 
argument we get again Pr^'[B{j)] < (1 — p^m)"' - ^ow we proceed exactly as in 
the proof of Theorem[2l substituting Pr[-] by Pr^[-], Ic by Iq, and S{wi, . . . wi) 
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by S{Ru. . . , R^), and obtain Pr^ila] = Pr^[^\^>o S{Ri, ■ • ■ , Rz)]- We conclude 
Pr'5[Runs(<P)] 

= Pr^[Runs(Xp) \ Ig] + Pr'^if] . . . , i?,)] 

= Pr^[Runs(7Wp) \ la] + Pr^^ila] 
= 1. 

Proof of Part (2): The proof proceeds analogously to the one of Theorem [31 
part (2): Let cti, (T2, ■ • ■ be a (countable or infinite) enumeration of the nodes in /. 
With Part (3) we obtain for each i > 1 a response Ri such that {AC)* Ri{AC)'^ 
is a terminating pattern for P, if the only starting node considered is at. By its 
definition, the universal pattern is a subset of {AC)* Ri{AC)" for every i > 1, 
so it is also terminating. 

Proof of Part (3): We reintroduce several notations from the proof of The- 
orem [2] and generalize them to accomodate nondeterminism. We say now that 

q £ Q ends up in q' £ Q following w — x\Xi . . . Xm G G* if 

q > q , 

and g' is probabilistic, nondeterministic, or T. Again, if such a g' exist, it is 
unique, since all transition choices are resolved. 

For every reachable node q € Qa and every sequence w G {AC)* holds that 
either: (i) q ends up in a node following w, or (ii) q ends up in T following 
a prefix of w. Otherwise there exists a node g' from which no probabilistic or 
nondeterministic location or T is reachable any more, which contradicts that P 
is a. s. -terminating (note that there always exists a strategy that is able to cause 
Qo ending up in g' with nonzero probability, using the nondeterministic choices 
given in see also below). 

We show that for every node q G Qa and every sequence si . . . s„ G A" there 
exists a c\C2 ■ ■ - Cn such that q ends up in T following a prefix of sici . . . s„c„. 
Assume for the sake of contradiction that there exists q G Qa and a sequence 
si . . . Sn G A" such that no ci . . . c„ exists with the property described above. 
We will construct a strategy S such that (i) reaching q has probability > 0, (ii) 
every run reaching q will never reach T. The probability of reaching T is then 
smaller than 1, contradicting the assumption that P is a. s. terminating. Recall 
that nodes of 7Wp[iS'] are paths in Mp. Since q is reachable in M-p, there exists 
a cycle-free path tt from the initial node qo to q. For all proper path prefixes of tt 
ending in a nondeterministic node, 5* selects the corresponding choices contained 
in TT with probability 1, and thus we reach q with probability > 0. For (ii), let 

TT be a path having the form tt = tt' — ^ qi q2 . . . qm, with m > 1 
and qi = q, such that tt' does not contain q. We define 5(7r) as follows: Let tt^. 

be the path obtained from qi ^ q2 ^ . . . qm by removing all possible 
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cycles. 7Tr then contains k < n nondeterministic nodes (there are only n nodes in 
total). Set S{TT){sk) = 1. Then there is no path starting from a reachable node 
tt' — >■ g in 7Vlp[S'] that reaches T (more exactly, that reaches a node tt" — >• T), 
contradicting the assumption that P is a.s.-terminating. 

We now select a ci . . . c,i e C" with the property described above for each q e Qa 
and si . . . s„ e A"-, and define tr{q, si . . . s„) := sici . . . s„c„. We set 

R{q) := {tr(g,w;) | w G A"}. 

Note that every R{q) is a response, and for every w G R{q), q ends up in T 
following a prefix of w. We say that a response with this property leads q to T. 
We construct now a sequence R'^^\R'^^\ . . . , i^^™) using the following algorithm. 
Set := {e} and i := 1. 

1. Pick a, q'i G Qa that does end up in a node qt following a w G R^^~^\ If 
no such qi exists set R := R^''~^'> and terminate. 

2. Set := (i?(*-i) \ {w}) Uwo R{q,). Set i := i + 1 and go to 1). 

We show that for every i, ii w G R^^\ \w\ < n^. This implies termination of the 
algorithm. 

Let w £ R'^^\ Let g^, . . . , be the nodes selected in part (1) of the algorithm 
such that w = W1W2 ■ ■ -Wm with Wj G Riq'j) for 1 < j < m. We define a family 
of node sets by: 

- Q(0) = QaU{T}, 

— for every j > 1, Q{j) is the set of nodes consisting of T and all nodes q such 
that a g G Qa ends up in q following wi. ..Wj. 

For every j > 0, |Q(j)| > 1- Wc now prove that Q{j) contains at most n — j 
nodes. This is true for Q(0). For j > 0, note that q'j is chosen such that wj or 
one of its prefixes leads a g G Q{j — 1) to T. That implies \Q{j)\ < \Q{j — 
and therefore the property (recall that every node ends up in at most one node 
following a sequence). 

Thus m has to be smaller than n — 1, and \w\ < n^, since R{q) has length n for 
all q. Note that for every w,w' G i?*^'^ for all i > 0, if w =/= w' then 'w\a 7^ w'\a- 
Hence after termination of the procedure, we can replace every w £ R such that 
\w\ = k-n Kn^ hy wo R', with R' an arbitrary response of length [n — k)- n, to 
obtain equal length of all words in i?, which then forms a response of length 11? . 
For every w £ R, every node of Qa ends up in T after following a prefix of w. 
We can conclude that every run r with fjc a prefix of a word in {AC)*R{AC)" 
is terminating, and thus {AC)*R{AC)" is a terminating pattern. □ 



